<!DOCTYPE html>
<html lang="zh-CN">
<head>
    <meta charset="UTF-8">
<meta name="viewport"
      content="width=device-width, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">

    <meta name="author" content="豆豆">


    <meta name="subtitle" content="愿望是实现睡觉自由">


    <meta name="description" content="你来啦，这里是豆豆的小笔记！">


    <meta name="keywords" content="豆豆,学习,小笔记,睡觉自由">


<title>「AI_04」回归演示 | 豆豆的小笔记</title>



    <link rel="icon" href="/favicon.ico">




    <!-- stylesheets list from _config.yml -->
    
    <link rel="stylesheet" href="/css/style.css">
    



    <!-- scripts list from _config.yml -->
    
    <script src="/js/script.js"></script>
    
    <script src="/js/tocbot.min.js"></script>
    



    
    
        
            <!-- MathJax配置，可通过单美元符号书写行内公式等 -->
<script type="text/x-mathjax-config">
    MathJax.Hub.Config({
    "HTML-CSS": {
        preferredFont: "TeX",
        availableFonts: ["STIX","TeX"],
        linebreaks: { automatic:true },
        EqnChunk: (MathJax.Hub.Browser.isMobile ? 10 : 50)
    },
    tex2jax: {
        inlineMath: [ ["$", "$"], ["\\(","\\)"] ],
        processEscapes: true,
        ignoreClass: "tex2jax_ignore|dno",
        skipTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code']
    },
    TeX: {
        equationNumbers: { autoNumber: "AMS" },
        noUndefined: { attributes: { mathcolor: "red", mathbackground: "#FFEEEE", mathsize: "90%" } },
        Macros: { href: "{}" }
    },
    messageStyle: "none"
    });
</script>
<!-- 给MathJax元素添加has-jax class -->
<script type="text/x-mathjax-config">
    MathJax.Hub.Queue(function() {
        var all = MathJax.Hub.getAllJax(), i;
        for(i=0; i < all.length; i += 1) {
            all[i].SourceElement().parentNode.className += ' has-jax';
        }
    });
</script>
<!-- 通过连接CDN加载MathJax的js代码 -->
<script type="text/javascript" async
        src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-MML-AM_CHTML">
</script>


        
    


<meta name="generator" content="Hexo 4.2.0"></head>
<body>
    <div class="wrapper">
        <header>
    <nav class="navbar">
        <div class="container">
            <div class="navbar-header header-logo"><a href="/">豆豆的小笔记</a></div>
            <div class="menu navbar-right">
                
                    <a class="menu-item" href="/archives">归档</a>
                
                    <a class="menu-item" href="/category">分类</a>
                
                    <a class="menu-item" href="/tag">标签</a>
                
                    <a class="menu-item" href="/about">关于</a>
                
                <input id="switch_default" type="checkbox" class="switch_default">
                <label for="switch_default" class="toggleBtn"></label>
            </div>
        </div>
    </nav>

    
    <nav class="navbar-mobile" id="nav-mobile">
        <div class="container">
            <div class="navbar-header">
                <div>
                    <a href="/">豆豆的小笔记</a><a id="mobile-toggle-theme">·&nbsp;纯白</a>
                </div>
                <div class="menu-toggle" onclick="mobileBtn()">&#9776; 菜单</div>
            </div>
            <div class="menu" id="mobile-menu">
                
                    <a class="menu-item" href="/archives">归档</a>
                
                    <a class="menu-item" href="/category">分类</a>
                
                    <a class="menu-item" href="/tag">标签</a>
                
                    <a class="menu-item" href="/about">关于</a>
                
            </div>
        </div>
    </nav>

</header>
<script>
    var mobileBtn = function f() {
        var toggleMenu = document.getElementsByClassName("menu-toggle")[0];
        var mobileMenu = document.getElementById("mobile-menu");
        if(toggleMenu.classList.contains("active")){
           toggleMenu.classList.remove("active")
            mobileMenu.classList.remove("active")
        }else{
            toggleMenu.classList.add("active")
            mobileMenu.classList.add("active")
        }
    }
</script>
        <div class="main">
            <div class="container">
    
    

    
    <article class="post-wrap">
        <header class="post-header">
            <h1 class="post-title">「AI_04」回归演示</h1>
            
                <div class="post-meta">
                    

                    
                        <span class="post-time">
                        时间： <a href="#">三月 13, 2017&nbsp;&nbsp;18:40:57</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                        </span>
                    
                    
                        <span class="post-category">
                    分类：
                            
                                <a href="/categories/AI/">AI</a>
                            
                        </span>
                    
                </div>
            
        </header>

        <div class="post-content">
            <p>现在假设有10个<code>x_data</code>和<code>y_data</code>，<code>x</code>和<code>y</code>之间的关系是<code>y_data=b+w*x_data</code>。<code>b</code>，<code>w</code>都是参数，是需要学习出来的。现在我们来练习用梯度下降找到<code>b</code>和<code>w</code>。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="keyword">from</span> pylab <span class="keyword">import</span> mpl</span><br><span class="line"></span><br><span class="line"><span class="comment"># matplotlib没有中文字体，动态解决</span></span><br><span class="line">plt.rcParams[<span class="string">'font.sans-serif'</span>] = [<span class="string">'Simhei'</span>]  <span class="comment"># 显示中文</span></span><br><span class="line">mpl.rcParams[<span class="string">'axes.unicode_minus'</span>] = <span class="literal">False</span>  <span class="comment"># 解决保存图像是负号'-'显示为方块的问题</span></span><br></pre></td></tr></table></figure>

<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">x_data = [<span class="number">338.</span>, <span class="number">333.</span>, <span class="number">328.</span>, <span class="number">207.</span>, <span class="number">226.</span>, <span class="number">25.</span>, <span class="number">179.</span>, <span class="number">60.</span>, <span class="number">208.</span>, <span class="number">606.</span>]</span><br><span class="line">y_data = [<span class="number">640.</span>, <span class="number">633.</span>, <span class="number">619.</span>, <span class="number">393.</span>, <span class="number">428.</span>, <span class="number">27.</span>, <span class="number">193.</span>, <span class="number">66.</span>, <span class="number">226.</span>, <span class="number">1591.</span>]</span><br><span class="line">x_d = np.asarray(x_data)</span><br><span class="line">y_d = np.asarray(y_data)</span><br></pre></td></tr></table></figure>


<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">x = np.arange(<span class="number">-200</span>, <span class="number">-100</span>, <span class="number">1</span>)</span><br><span class="line">y = np.arange(<span class="number">-5</span>, <span class="number">5</span>, <span class="number">0.1</span>)</span><br><span class="line">Z = np.zeros((len(x), len(y)))</span><br><span class="line">X, Y = np.meshgrid(x, y)</span><br></pre></td></tr></table></figure>

<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># loss</span></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(len(x)):</span><br><span class="line">    <span class="keyword">for</span> j <span class="keyword">in</span> range(len(y)):</span><br><span class="line">        b = x[i]</span><br><span class="line">        w = y[j]</span><br><span class="line">        Z[j][i] = <span class="number">0</span>  <span class="comment"># meshgrid吐出结果：y为行，x为列</span></span><br><span class="line">        <span class="keyword">for</span> n <span class="keyword">in</span> range(len(x_data)):</span><br><span class="line">            Z[j][i] += (y_data[n] - b - w * x_data[n]) ** <span class="number">2</span></span><br><span class="line">        Z[j][i] /= len(x_data)</span><br></pre></td></tr></table></figure>

<p>先给<code>b</code>和<code>w</code>一个初始值，计算出<code>b</code>和<code>w</code>的偏微分</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># linear regression</span></span><br><span class="line"><span class="comment">#b = -120</span></span><br><span class="line"><span class="comment">#w = -4</span></span><br><span class="line">b=<span class="number">-2</span></span><br><span class="line">w=<span class="number">0.01</span></span><br><span class="line">lr = <span class="number">0.000005</span></span><br><span class="line">iteration = <span class="number">1400000</span></span><br><span class="line"></span><br><span class="line">b_history = [b]</span><br><span class="line">w_history = [w]</span><br><span class="line">loss_history = []</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line">start = time.time()</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(iteration):</span><br><span class="line">    m = float(len(x_d))</span><br><span class="line">    y_hat = w * x_d  +b</span><br><span class="line">    loss = np.dot(y_d - y_hat, y_d - y_hat) / m</span><br><span class="line">    grad_b = <span class="number">-2.0</span> * np.sum(y_d - y_hat) / m</span><br><span class="line">    grad_w = <span class="number">-2.0</span> * np.dot(y_d - y_hat, x_d) / m</span><br><span class="line">    <span class="comment"># update param</span></span><br><span class="line">    b -= lr * grad_b</span><br><span class="line">    w -= lr * grad_w</span><br><span class="line"></span><br><span class="line">    b_history.append(b)</span><br><span class="line">    w_history.append(w)</span><br><span class="line">    loss_history.append(loss)</span><br><span class="line">    <span class="keyword">if</span> i % <span class="number">10000</span> == <span class="number">0</span>:</span><br><span class="line">        print(<span class="string">"Step %i, w: %0.4f, b: %.4f, Loss: %.4f"</span> % (i, w, b, loss))</span><br><span class="line">end = time.time()</span><br><span class="line">print(<span class="string">"大约需要时间："</span>,end-start)</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># plot the figure</span></span><br><span class="line">plt.contourf(x, y, Z, <span class="number">50</span>, alpha=<span class="number">0.5</span>, cmap=plt.get_cmap(<span class="string">'jet'</span>))  <span class="comment"># 填充等高线</span></span><br><span class="line">plt.plot([<span class="number">-188.4</span>], [<span class="number">2.67</span>], <span class="string">'x'</span>, ms=<span class="number">12</span>, mew=<span class="number">3</span>, color=<span class="string">"orange"</span>)</span><br><span class="line">plt.plot(b_history, w_history, <span class="string">'o-'</span>, ms=<span class="number">3</span>, lw=<span class="number">1.5</span>, color=<span class="string">'black'</span>)</span><br><span class="line">plt.xlim(<span class="number">-200</span>, <span class="number">-100</span>)</span><br><span class="line">plt.ylim(<span class="number">-5</span>, <span class="number">5</span>)</span><br><span class="line">plt.xlabel(<span class="string">r'$b$'</span>)</span><br><span class="line">plt.ylabel(<span class="string">r'$w$'</span>)</span><br><span class="line">plt.title(<span class="string">"线性回归"</span>)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure>
<p>输出结果如图</p>
<p><img src="chapter4-1.png" alt=""></p>
<p>横坐标是<code>b</code>，纵坐标是<code>w</code>，标记<code>*</code>为最优解，显然，在图中我们并没有运行得到最优解，最优解十分的遥远。那么我们就调大<code>learning rate</code>，<code>lr = 0.000001</code>（调大10倍），得到结果如下图。</p>
<p><img src="chapter4-2.png" alt=""></p>
<p>我们再调大<code>learning rate</code>，<code>lr = 0.00001</code>（调大10倍），得到结果如下图。</p>
<p><img src="chapter4-3.png" alt=""></p>
<p>结果发现<code>learning rate</code>太大了，结果很不好。</p>
<p>所以我们给<code>b</code>和<code>w</code>特制化两种<code>learning rate</code>：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># linear regression</span></span><br><span class="line">b = <span class="number">-120</span></span><br><span class="line">w = <span class="number">-4</span></span><br><span class="line">lr = <span class="number">1</span></span><br><span class="line">iteration = <span class="number">100000</span></span><br><span class="line"></span><br><span class="line">b_history = [b]</span><br><span class="line">w_history = [w]</span><br><span class="line"></span><br><span class="line">lr_b=<span class="number">0</span></span><br><span class="line">lr_w=<span class="number">0</span></span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line">start = time.time()</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(iteration):</span><br><span class="line">    b_grad=<span class="number">0.0</span></span><br><span class="line">    w_grad=<span class="number">0.0</span></span><br><span class="line">    <span class="keyword">for</span> n <span class="keyword">in</span> range(len(x_data)):</span><br><span class="line">        b_grad=b_grad<span class="number">-2.0</span>*(y_data[n]-n-w*x_data[n])*<span class="number">1.0</span></span><br><span class="line">        w_grad= w_grad<span class="number">-2.0</span>*(y_data[n]-n-w*x_data[n])*x_data[n]</span><br><span class="line">    </span><br><span class="line">    lr_b=lr_b+b_grad**<span class="number">2</span></span><br><span class="line">    lr_w=lr_w+w_grad**<span class="number">2</span></span><br><span class="line">    <span class="comment"># update param</span></span><br><span class="line">    b -= lr/np.sqrt(lr_b) * b_grad</span><br><span class="line">    w -= lr /np.sqrt(lr_w) * w_grad</span><br><span class="line"></span><br><span class="line">    b_history.append(b)</span><br><span class="line">    w_history.append(w)</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># plot the figure</span></span><br><span class="line">plt.contourf(x, y, Z, <span class="number">50</span>, alpha=<span class="number">0.5</span>, cmap=plt.get_cmap(<span class="string">'jet'</span>))  <span class="comment"># 填充等高线</span></span><br><span class="line">plt.plot([<span class="number">-188.4</span>], [<span class="number">2.67</span>], <span class="string">'x'</span>, ms=<span class="number">12</span>, mew=<span class="number">3</span>, color=<span class="string">"orange"</span>)</span><br><span class="line">plt.plot(b_history, w_history, <span class="string">'o-'</span>, ms=<span class="number">3</span>, lw=<span class="number">1.5</span>, color=<span class="string">'black'</span>)</span><br><span class="line">plt.xlim(<span class="number">-200</span>, <span class="number">-100</span>)</span><br><span class="line">plt.ylim(<span class="number">-5</span>, <span class="number">5</span>)</span><br><span class="line">plt.xlabel(<span class="string">r'$b$'</span>)</span><br><span class="line">plt.ylabel(<span class="string">r'$w$'</span>)</span><br><span class="line">plt.title(<span class="string">"线性回归"</span>)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure>

<p><img src="chapter4-4.png" alt=""></p>
<p>有了新的特制化两种<code>learning rate</code>就可以在10w次迭代之内到达最优点了。</p>

        </div>

        
        <section class="post-tags">
            <div>
                <span>标签：</span>
                <span class="tag">
                    
                    
                        <a href="/tags/%E6%9D%8E%E5%AE%8F%E6%AF%85/"># 李宏毅</a>&nbsp;&nbsp;
                    
                        <a href="/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"># 机器学习</a>&nbsp;&nbsp;
                    
                        <a href="/tags/AI/"># AI</a>&nbsp;&nbsp;
                    
                        
                </span>
            </div>
            <div>
                <a href="javascript:window.history.back();">返回</a>
                <span>· </span>
                <a href="/">主页</a>
            </div>
        </section>
        <section class="post-nav">
            
                <a class="prev" rel="prev" href="/2017/03/13/ai-lhyML-05/">「AI_05」误差的来源</a>
            
            
            <a class="next" rel="next" href="/2017/03/12/ai-lhyML-03/">「AI_03」回归</a>
            
        </section>


    </article>
</div>

        </div>
        <footer id="footer" class="footer">
    <div class="copyright">
        <span>© 豆豆 | Powered by <a href="https://hexo.io" target="_blank">Hexo</a> & <a href="https://github.com/Siricee/hexo-theme-Chic" target="_blank">Chic</a></span>
    </div>
</footer>


    <script src='https://unpkg.com/mermaid@7.1.2/dist/mermaid.min.js'></script>
    <script>
      if (window.mermaid) {
        mermaid.initialize({theme: 'neutral'});
      }
    </script>
  
    </div>
</body>
</html>
